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Executive Summary 


This report presents results from a study of the Council of State Science Supervisors (CSSS). 
CSSS is a professional association for state science leaders that works to sustain and nurture a 
dynamic learning community and to empower its members to be effective and articulate 
advocates for quality science education at the local, state, and national levels. Given that many 
state science supervisors are working to support implementation of the Framework for K-12 
Science Education, we sought to develop an understanding of CSSS members’ roles and 
responsibilities, as well as their use of research to inform state implementation decisions. While 
state education agencies play key roles in supporting the implementation of standards that align 
with the Framework, with state science supervisors taking a lead role, little is known about how 
these individuals use research to inform their decisions. 


The research questions guiding the study were: 

e What roles and activities do CSSS members take on and participate in within the 
association and their states? 

e What research findings and research-based resources do CSSS members use to inform 
their efforts to promote implementation of the Framework? What research findings and 
research-based resources do they share with local education leaders? 

e Towhom do CSSS members turn for research to inform their state’s decisions regarding 
implementation of the Framework? 

e How, if at all, are CSSS member roles and activities associated with their research 
networks? 


Research Design and Methods 


We administered a survey focused on the roles that CSSS members take on in the organization, 
the state-level activities in which they participate, how they use research, and to whom they turn 
for research to inform decisions related to statewide implementation of the vision for science 
teaching and learning described in the Framework. While survey items pertaining to CSSS 
members roles and activities were developed in collaboration with CSSS members,? items 
related to research use were previously developed and validated via a national survey conducted 
by the National Center for Research in Policy and Practice (NCRPP).3 Items related to CSSS 
members’ research networks (i.e., to whom they turn for research) drew on prior social network 
studies‘ and researcher expertise. 


The survey was administered twice at the annual CSSS conference, once in April 2016 and once 
in April 2017. The data were collected just after CSSS concluded a sustained professional 
development project for state science supervisors (Building Capacity in State Science Education 
or BCSSE) and before it began a research-practice partnership to improve equity by building 
coherence in science education within and between states (Advancing Coherent and Equitable 
Systems of Science Education or ACESSE). 
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This report primarily focuses on results from the second survey administration in April 2017,5 
with significant changes between years reported as applicable. In spring 2017, a total of 58 
individuals responded to the survey from 36 states; these individuals included state members 
(n=40), associate and honorary members (n=11), and affiliate members (n=7) of CSSS. 


The overall response rate for the survey in 2017 was 41%, compared to 62% in 2016. The lower 
response rate in 2017 can be attributed to: 1) the inclusion of a more diverse population 
representing different kinds of CSSS members (defined below); and 2) membership turnover, as 
the majority of individuals (i.e., 14 of 20) who responded in 2016 but not in 2017 left their state- 
level positions or no longer belonged to CSSS. On average, respondents in 2017 had four years of 
experience as state science leaders, and the majority of the sample was white and female. 


Definition of Research and CSSS Membership Used in the Study 


For the purposes of this study, we defined “research” as an activity in which people employ 
systematic, empirical methods to answer a specific question. In this sense, research is different 
than the practice of looking at data from the district, school, or classroom, which is more open- 
ended and seldom addresses specific research questions. 


We delineated four types of CSSS members: (1) State members, who are employees of their state 
Department of Education and are designated by their chief state school officer as having 
responsibility for science education statewide; (2) Associate members, who are former state 
members; (3) Affiliate members, who are interested in the vision, mission, purposes, and goals 
of CSSS and are often engaged in research related to science education; and (4) Honorary life 
members, who are nominated by the CSSS Board and elected by a majority of members for their 
significant service to CSSS and/or to science education. Given their prior history as state 
leaders, we combined associate and honorary members in our analyses. 


Findings Related to Roles and Activities 


CSSS roles. The survey asked respondents about the roles they have assumed within CSSS. In 
response to these items, CSSS members most often reported serving as conference presenters or 
participants, at both their state science conferences and the CSSS annual meeting. As compared 
to other activities, they reported less frequent engagement in CSSS leadership activities, such as 
committee and board meetings. Overall, associate, honorary, and affiliate members were more 
often engaged in out-of-state or national activities than state members. 


CSSS activities. Respondents were asked to report how often they participated in a range of 
CSSS activities over the last three years. Overall, respondents reported frequently accessing 
information from the CSSS listserv, as well as participating in CSSS-sponsored webinars, 
consulting with CSSS members, and collaborating with other states. State members were 
significantly less likely to report visiting other states and presenting at national meetings than 
associate, honorary, and affiliate members. This difference may be related to variation in the 
roles and responsibilities of state and non-state members. 
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State activities. Respondents also reported engagement in a variety of state-level activities in 
the areas of curriculum, instruction, assessment, and professional development. They reported 
being highly involved in state policy decisions related to science standards, yet they were less 
involved with the implementation of curricula that reflected those standards. They also reported 
playing key roles in assessment and professional development, although few respondents had 
authority to make decisions related to resource allocation or contract selection for assessment 
systems or professional development providers. 


Professional development. Based on a request from CSSS leaders, the 2017 survey included 
items related to teacher professional development (PD). Respondents were asked to identify the 
PD offered in their state that afforded teachers the best opportunity to learn about the 
Framework. Findings revealed that these opportunities tended to be led by a state agency or 
local leader and funded by federal grant programs. These PD opportunities covered 
foundational concepts such as the three dimensions of science learning in the Framework, Next 
Generation Science Standards (NGSS), or instructional planning; however, they tended to pay 
less attention to designing three-dimensional assessments and developing students’ skills. 


Findings Related to CSSS Member Research Use 


Specific pieces of research found useful or shared with others. The survey asked 
respondents to name a specific piece of research they found useful for informing their state’s 
decisions related to implementation of the Framework, as well as a piece of research they 
shared with district or school leaders. The research respondents used to support 
implementation of the Framework most often focused on student learning and classroom 
assessment, while the research they shared with local leaders most often focused on classroom 
assessment and pedagogical practices. Few respondents named research focused on the needs or 
assets of particular student subgroups (e.g., by race/ethnicity, gender, socioeconomic status, or 
language). Overall, the pieces of research named were primarily research reports or policy 
briefs, particularly those published by the National Academies of Science, Engineering, and 
Medicine, as well as peer-reviewed journal articles. 


Research trustworthiness. Respondents were asked to indicate why they found the 
particular pieces of research they named trustworthy. The most often cited reasons were if the 
research findings applied to their state context, or if the research gave them new ideas to 
support implementation of the Framework. A less commonly cited reason was that the research 
methods were rigorous. 


Sources used to obtain research. The survey asked respondents how often they obtained 
research from a list of 13 sources. Of these, many respondents indicated obtaining research 
through CSSS or colleagues in their state departments of education. Far fewer respondents 
indicated seeking out research from the National Science Education Leadership Association 
(NSELA) or from the What Works Clearinghouse (WWC). 
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Efforts to acquire research. We asked respondents to indicate whether they would seek out 
research under different conditions. Although a majority said they would look for research to 
inform a new problem or decision, fewer said they would contact researchers directly under 
these circumstances, especially researchers they did not already know. 


Findings Related to Research Networks 


Research networks. The survey asked CSSS members to whom they have turned for research 
to inform their state’s implementation of the Framework. Findings from these social network 
questions revealed that associate, honorary, and affiliate members served as prominent sources 
of research in the areas of curriculum, instruction, assessment, and professional development. 
University faculty not affiliated with CSSS were also frequently named as sources of research. 
Additionally, new state members were more likely than veteran state members to name 
individuals from their own states as sources of research. And, while some respondents 
facilitated the exchange of research within the professional association, or between CSSS 
members in different states, others facilitated the exchange of research between CSSS members 
and researchers unaffiliated with the association. 


Relations between activities and networks. Our Year 1 report indicated that an important 
next step in this study was to explore relationships between CSSS members’ roles and activities 
and their use of research. We used Year 2 data to examine these relationships, and found that 
participation in structured CSSS activities, particularly substantive meetings as compared to 
planning meetings or more informal interactions such as webinars, were important for 
facilitating the exchange of research among CSSS members. 
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Introduction 


The core studies conducted by the National Center for Research in Policy and Practice (NCRPP) 
focus on research use among school and district leaders, but leaders in state education agencies 
also make use of research to inform their decision making. Contrary to many depictions of state 
agencies, state education agency leaders have many social ties across departments and with 
external organizations, ties along which research flows.° Nonetheless, research use among state 
education agency leaders remains a largely understudied phenomenon. Given that professional 
associations can be an important source of social support and research-based resources that 
help state education agency leaders do their jobs effectively,” our study focuses on the lead 
professional association for state leaders in science education, the Council of State Science 
Supervisors (CSSS). 


CSSS is a professional organization composed of science education specialists who serve at the 
state, territorial, or the protectorate educational agency in the United States and U.S. 
Territories. Its members primarily include current supervisors of science education in state 
education agencies. Within their states or jurisdictions, these supervisors play key roles in 
directing efforts at improving school science instruction and in ensuring excellence and equity in 
science education. As a professional organization, CSSS organizes meetings, facilitates 
consultations between state leaders, provides learning opportunities to its members, and serves 
as a linkage point from outside organizations to schools and districts. Beyond current state 
supervisors, CSSS members include former supervisors (i.e., associate members) as well as a 
small number of honorary members and research affiliates. Overall, CSSS members are 
knowledgeable about state standards, curriculum, assessment systems, and professional 
development providers, as well as state-level initiatives to improve science education. 


One such initiative is the recent implementation of the vision of equitable science teaching and 
learning described in Framework for K-12 Science Education,’ which calls for significant 
changes to science education to ensure that all students become proficient in science. It calls for 
systems to be organized around building understanding of disciplinary core ideas over time, 
engagement of students in the practices of science and engineering, and application of 
crosscutting concepts that unify science. It also calls for science teaching to promote equity 
and—as one strategy for doing so—to connect learning opportunities to students’ everyday 
experiences, interests, and identities. It is based on a large body of research on how students 
best learn science? and on careful observations of the real work of scientists and engineers.'° In 
recent years, a number of states have chosen to adopt the Next Generation Science Standards" 
or standards based on the Framework, an important first step to reorganizing science education 
to achieve the ambitious aims laid out in the Framework. Additional changes to curriculum, 
instruction, teacher preparation and professional development, and student assessment will be 
required, especially to transform science education opportunities for an increasingly diverse 
student population.'? 
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Given that many state leaders in science education are working to support implementation of 
the Framework, we sought to develop an understanding of CSSS members’ work, as well as their 
use of research to inform their state’s implementation decisions. We asked: 


e What roles and activities do CSSS members take on and participate in within the 
association and their states? 

e What research findings and research-based resources do CSSS members use to inform 
their efforts to promote implementation of the Framework? What research findings and 
research-based resources do they share with local science education leaders? 

e Towhom do CSSS members turn for research to inform their state’s decisions regarding 
implementation of the Framework? 

e How, if at all, are CSSS member roles and activities associated with their research 
networks? 


In this report, we provide information related to the study design and sample, and findings on 
roles and activities in which respondents reported engaging as CSSS members and state leaders. 
We also present findings related to CSSS members’ research use, with respect to the sources 
they sought for research to inform their state’s decisions, as well as their efforts to acquire 
research. In addition, we investigate CSSS members’ social networks, and specifically how 
research-related information flows between members. We also examine relationships between 
characteristics of CSSS members’ networks and their activities in the association. Finally, we 
present key conclusions drawn from the survey data analysis. 
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Study Design 


In spring 2017, we surveyed a national sample of CSSS members to explore the questions 
outlined above. A similar version of this survey was administered to CSSS members in spring 
2016, with some adaptations made in 2017 based on consultation with CSSS leaders. 
Additionally, there were some key differences in the population surveyed and resulting sample 
between 2016 and 2017 (which we refer to as Year 1 and Year 2, respectively), as well as the data 
collection procedures. We describe these below, as well as the survey instrument and sample. 


Population 


Our population included four types of CSSS members, who we placed into three groups. First, 
state members are designated by their chief state school officers as having responsibility for 
science education statewide. Second, associate and honorary members are former state 
members and are recognized by the CSSS board for their service to the organization. Third, 
affiliate members are individuals interested in the vision and mission of CSSS and are often 
engaged in science education research. To identify the population for this study, we used a 
membership list provided by CSSS leaders that included 78 state members, 42 associate and 
honorary members, and 25 affiliate members. 


Overall, the 145 CSSS members invited to participate in spring 2017 represented 47 states. In 
terms of membership, this group represented a more diverse population than in spring 2016, 
when the population (n=98) primarily included state members and a small number of associate 
and honorary members. This difference in population was intentional, as we targeted state 
members in Year 1, and expanded our focus in Year 2. 


Data Collection Procedures 


We collected surveys from CSSS members in two ways. First, we distributed the survey at the 
2017 Council of State Science Supervisors Annual Conference. All conference attendees who 
were willing to participate filled out a paper survey, due to the lack of Internet access at the 
conference center. Second, we sent email messages to any individuals on the membership list 
who had not yet completed the survey and invited them to complete an online version via 
Qualtrics. A maximum of three follow-up invitations were sent to these individuals over a one- 
month period. This process was very similar in Year 1, except that in Year 1 conference attendees 
were able to complete the survey online rather than on paper. 


Survey Instrument 


The survey instrument was developed to address our four research questions. Items related to 
CSSS members’ roles and activities were developed in collaboration with CSSS members. 
Another important set of items was drawn from the previously developed and validated items 
included in the national survey conducted by the National Center for Research in Policy and 
Practice (NCRPP).!3 The social network items were developed based on previous studies as well 
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as researcher expertise. The survey instrument is shown in Appendix A. The definition of all 
survey constructs, sample items, item response choices, and the total number of items for each 
construct are provided below. 


CSSS roles. This item referred to the roles participants have taken on as CSSS members. We 
developed a list of nine roles in consultation with CSSS board members, such as: president, 
secretary, board member, and ad-hoc committee chair. Respondents could choose more than 
one role and indicate “other” as necessary. 


CSSS activities. These items asked participants to report the frequency with which they 
participated in association activities in the past three years. A list of activities was generated 
from a review of documents provided by the CSSS Board, then reviewed by Board members. 
Examples of activities include: annual CSSS meeting participation, consultations with other 
CSSS members, participation in CSSS-sponsored webinars, and attendance at workshops/talks 
by researchers. For each activity, respondents were asked how often they participated in the 
activity, with response choices as follows: Never (1), Once (2), 2-3 times (3), 4 or more times (4). 


State leader activities. These items pertain to the activities participants engaged in as part of 
their work as state science supervisors. In consultation with CSSS board members, we developed 
a list of such activities across five areas: standards and curriculum (e.g., reviewing state science 
standards), assessments (e.g., designing state assessments), professional development (e.g., 
writing contracts for professional development providers), partnerships (e.g., identifying 
resources to share with districts), and awards (e.g., coordinating student scholarships). 
Participants were asked to indicate how often they engaged in each activity in the last 12 
months. Item response choices for each activity were: Never (1), Rarely (2), Sometimes (3), 
Often (4), All the time (5). 


Teacher professional development. At the request of CSSS leaders, we included five 
survey items in Year 2 pertaining to teacher professional development (PD). We prompted 
respondents to think about the PD offered in their state in the last year that they thought offered 
the best opportunity for teachers to learn about the Framework for K-12 Science Education. We 
asked: (1) who led the PD; (2) how much time teachers spent together in the PD; (3) the period 
of time over which the PD occurred; and (4) whether particular topics were a minor or major 
focus of the PD. We also asked about the funding sources (e.g., Title II programs, School 
Improvement Grants) used to support teacher PD specific to science education. 


Research CSSS members found useful. Following an approach used in the national 
NCRPP survey, we sought to identify individual pieces of research that CSSS members found 
useful in informing decisions in their state related to implementation of the Framework. For 
each study named, we asked them to identify (if they could) the title, author, year published, 
publisher, topic, and why they found it useful. We augmented this series of items in Year 2 by 
asking respondents to indicate why they found that piece of research trustworthy, with response 
options including statements such as: “I was involved in the research,” “The research methods 
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used are rigorous,” “I could relate the findings to my state context,” and “It gave me new ideas 
for how to support implementation of the Framework.” 


Research CSSS members shared with others. We sought to identify individual pieces of 
research that CSSS members shared with district or school leaders related to implementation of 
the Framework. For each study named, we asked them to identify (if they could) the title, 
author, year published, publisher topic, and with whom they shared it. 


Research acquisition effort. This construct refers to the extent to which an individual exerts 
effort to acquire research and develop relationships with researchers to address problems or 
decisions.'4 The scale comprises five items developed by NCRPP, and asked CSSS members to 
indicate how often they engaged in certain activities, such as looking for research studies or 
contacting researchers to find out more about articles they have written. Item response choices 
were: Never (1), Rarely (2), Sometimes (3), Often (4), All the time (5). Analysis suggested 
moderate internal consistency for the Year 2 sample (n=58, a = 0.72), which was similar to the 
NCRPP national survey (n=73, a = 0.79).15 


Sources where CSSS members obtain research. These items pertain to the sources where 
CSSS members obtain research, and the frequency with which they consult those sources. 
Sources included traditional ones, such as university researchers, as well as networks (e.g., 
professional associations such as the National Science Teachers Association) and media. For 
each source, we asked how often individuals sought out or acquired research in the past 12 
months. Response choices were: Never (1), Rarely (2), Sometimes (3), Often (4), All the time (5). 


Research networks. These items sought to identify to whom CSSS members turned in the 
past 12 months for research to inform their state’s efforts to implement the Framework. First, 
participants named up to 12 researchers, state science supervisors, and other colleagues, either 
within or outside their states. Participants were then asked to indicate how often they turned to 
each person in the last year, considering all forms of communication such as face-to-face, e- 
mail, or telephone, with the following response options: 1-2 times per year (1), 3-4 times per 
year (2), Every 2 months or so (3), Monthly (4), 2-3 times per month (5), and weekly (6). Then, 
participants were asked what research topics they discussed with each person: curriculum, 
assessment, professional development, and instruction. 


Sample 


Of the 145 individuals invited to complete the survey in spring 2017, 58 completed the majority 
or all of the survey, for an overall response rate of 41%. Of the 48 states in the study population, 
36 states, or 75%, were represented in the sample. Whereas 49 respondents completed the paper 
survey at the CSSS Annual Meeting, 12 respondents completed the survey via Qualtrics. The 
sample comprised 40 state members, 11 associate or honorary members, and 7 affiliate 
members, who represented 51%, 26%, and 28%, respectively, of the total individuals surveyed in 
their membership group. 
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Table 1 includes characteristics of the survey sample in 2017, with comparisons to the year prior. 
Although the response rate was lower in Year 2 than in Year 1, the sample was intentionally 
more diverse in terms of membership type in Year 2. Nonetheless, the sample remained 
predominantly female and white, and the average respondent had 4-5 years of experience as a 
state science supervisor. As shown in the table, 33 respondents completed the survey in both 


years. 


Table 1. Survey Respondent Characteristics, Year 1 and Year 2 


Year 1 Year 2 
Both Years 

(2016) (2017) 
Number of respondents 61 58 33 
Response rate 62% 41% - 
States represented 38 36 24 
Associate or honorary 5 1 5 
members 
Affiliate members 0 7 0 


Self-identified gender 


69% female 


62% female 


65% female 


Race/ethnicity 81% white 88% white 81% white 
Average experience as state 
science supervisor, if 5 years A years 6 years 
applicable 
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CSSS Roles and Activities 


Key Findings 
e Respondents were more likely to report engagement in conferences than in 
leadership activities associated with CSSS. 


e All respondents reported frequently accessing information from the CSSS listserv, 
as well as participating in CSSS-sponsored webinars, consulting with CSSS 
members, and collaborating with other states. However, state members were 
significantly less likely than associate, honorary, and affiliate members to report 
visiting other states and presenting at national meetings. 


e Respondents reported being highly involved in state policy decisions related to 
science standards, yet were less involved with the implementation of curricula that 
reflected those standards. Similarly, respondents reported playing key roles in 
assessment and professional development, but few had authority to make 
decisions related to resource allocation or contract selection for assessment 
systems or professional development providers. 


e The professional development that respondents identified as providing the best 
opportunity for teachers to learn about the Framework tended to be funded by 
federal grant programs, and to be led by a state agency or local leader. These PD 
opportunities covered foundational concepts such as instructional planning and 
the three dimensions of science learning in the Framework or Next Generation 
Science Standards (NGSS), yet paid less attention to designing three-dimensional 
assessments and developing students’ skills in science. 


In this section we report findings from the items related to CSSS members’ roles and activities, 
exploring the frequency of their self-reported involvement. Findings are organized in four 
groups: CSSS-related roles, CSSS-related activities, state-level activities (standards and 
curriculum, assessment, professional development, partnership, and awards), and the teacher 
professional development that respondents found most useful. 


Roles Related to CSSS Membership 


The CSSS conference was the most frequent activity in which respondents reported engaging, be 
it as participants or presenters. And, CSSS members were more likely to report engagement in 
conferences than in leadership activities associated with the professional association. For 
example, a full 80% of respondents in spring 2017 indicated participating in their state science 
conference, and 37% indicated presenting at a CSSS conference (see Figure 1). In contrast, just 
14% of respondents indicated having served as a CSSS board member. The proportion of 
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respondents who indicated taking on leadership roles within CSSS, including president, 
secretary, and board member, remained about the same between 2016 and 2017. However, a 
larger proportion of Year 2 respondents indicated presenting at the CSSS conference (37% in 
Year 2 compared to 18% in Year 1), participating in their state science conference (80% in Year 2 
compared to 67% Year 1), as well as organizing their state science fair (14% in Year 2 compared 
to 7% in Year 1). 


: : 80% 
Participant in state science conference 


67% 


: : 37% 
Presenter at this year’s CSSS conference 
18% 


P 37% 
Presenter at a previous CSSS conference 
21% 


29% 
Ad-hoc committee contributor/participant a 
‘O 


29% 
Ad-hoc committee chair i 


23% 


14% 
15% 


Board member 


. = : 14% 
Organizer of state science fair 


7% 


6% 
7% 


President 


2% 
2% 


Secretary 


m 2017 (n=49) ™ 2016 (n=61) 


Figure 1. CSSS-Related Roles Assumed by Survey Respondents, Year 1 and Year 2 


Activities Related to CSSS Membership 

With respect to the CSSS activities that respondents participated in over the last three years, 
reading information from the CSSS listserv was cited most frequently, with 85% of respondents 
reporting doing so four or more times (see Figure 2). This finding was consistent between Year 1 
and Year 2. Consulting with CSSS members, collaborating with other states, attending 
workshops or talks by researchers, and participating in CSSS Annual Meetings were also 
frequently reported activities in both years. 
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Reading information from CSSS listerv 
Consulting with CSSS members 
Collaborating with other states 
Attending workshops/talks by researchers 
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Participating in CSSS sponsored webinars 
CSSS Committee Meetings 
Reviewing drafts of NGSS 
Presenting at NRC or other national meetings 
Visiting other states 
BCSSE Meetings 
CSSS Board Meetings 
mNever “Once 2-3times m4times or more 
Figure 2. CSSS Activity Participation in the Last Three Years (n=54). 
Activities by member type. In general, associate, honorary, and affiliate members engaged 
significantly more often in out-of-state or national activities than state members (p<.05).'° 
Specifically, whereas only 19% of state members (n=38) reported visiting other states at least 2- 
3 times in the last three years, 50% of associate and honorary members?” (n=10) and 66% of 
affiliate members (n=6) reported visiting other states at that frequency. Moreover, state 
members were significantly less likely than other members (p<.o1) to report presenting at 
national meetings, such as those held by the National Academies of Sciences, Engineering, and 
Medicine, with just 22% of state members reporting engagement in such presentations at least 


2-3 times in the last three years, compared to 58% of associate and honorary members, and 67% 
of affiliate members. 
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State-Level Activities 


A total of 50 respondents reported their frequency of participation in various state-level 
activities during the previous year. These activities centered in five areas: (1) standards and 
curriculum, (2) assessment, (3) professional development, (4) partnerships, and (5) awards. 


Standards and curriculum. State science leaders tended to engage more often in standards 
development or review than in the adoption of curriculum. Fifty-three percent (n=25) of 
respondents indicated often or all the time serving in lead roles to review or develop state 
science standards, and 44% (n=20) indicated serving in support roles often or all the time (see 
Figure 3a). On the other hand, 37% of respondents (n=19) reported that they never consulted 
with curriculum companies on products in the design phase, and a full 59% of respondents 
(n=30) indicated never organizing state curriculum adoption. Thus, although state science 
leaders were involved in policy decisions related to state standards, they were less involved with 
the implementation of curriculum that reflected those standards. 


Reviewing or developing state science standards (in lead 
role) 


Reviewing or developing state science standards (in support 
role) 


Advising state committee's decisions on science course 
content and/or graduation policies (in lead role) 


Advising state committee's decisions on science course 
content and/or grad policies (in support role) 


Organizing state curriculum adoption 


Consulting with curriculum companies on products in the 
design phase 


mNever Rarely Sometimes mOften gAIll of the time 


Figure 3a. Frequency of Engagement in State-level Activities, Standards and Curriculum (n=50). 


Findings from a Survey of State Science Leaders: Year Two 17 


National Center for Research in Policy and Practice 


Assessment. Respondents were more likely to report involvement in designing state 
assessments than selecting contractors for state assessments. Specifically, 56% of respondents 
(n=28) indicated that they played a support role in designing state assessments often or all the 
time, while 36% (n=18) reported playing a lead role (see Figure 3b). Conversely, only 26% 
(n=13) reported selecting contractors for state assessments at that frequency. These findings 
indicate that many state science leaders played key roles in assessment, but few had authority to 
make decisions consequential for statewide testing. 


Designing state assessments (in support role) mm 9% 26% 
Designing state assessments (in lead role) ee 14% 
Sere re eer re i 


mNever Rarely Sometimes mOften maAIll of the time 


Figure 3b. Frequency of Engagement in State-level Activities, Assessment (n=50). 


Professional development. Respondents reported being highly involved in designing or 
conducting professional development in their states, and less involved in allocating resources or 
contacting PD providers. A full 77% (n=39) of respondents reported conducting professional 
development often or all the time (see Figure 3c). Further, 72% (n=37) indicated that they led 
the design of professional development often or all the time, while 62% (n=31) reported playing 
a support role at that frequency. On the other hand, respondents were less likely to report 
identifying vendors to provide PD, writing contracts for professional development, or allocating 
Title 2A funds, with just 24% (n=12), 26% (n=13), and 10% (n=5), respectively, reporting 
involvement in these activities often or all the time. As with assessment, these findings suggest 
that, although CSSS members were highly involved in state professional development, most had 
limited authority to allocate resources to support it. 
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Figure 3c. Frequency of Engagement in State-level Activities, Professional Development (n=50). 


The only significant difference in any reported activities between respondents in Year 1 and 2 
(n=33) was in their frequency of involvement with writing contracts for PD providers. In Year 2, 
respondents were significantly less likely to be involved in this activity (p<.05), which could be 
related to federal changes that ended Math Science Partnership (MSP) funding, which many 
state science supervisors had helped to manage. 


Partnerships. Respondents tended to engage more often in resource sharing with districts as 
opposed to establishing collaborations with their state university systems. They most frequently 
reported identifying resources to share with districts, or screening or reviewing resources to 
share with districts, with 80% (n=31) and 63% (n=32) of respondents reporting doing so often 
or all of the time, respectively (see Figure 3d). Then, 53% (n=27) reported establishing 
partnerships with business, industry, or non-formal groups often or all the time, and 49% 
(n=25) reported collaborating with state university systems at that frequency. 
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Figure 3d. Frequency of Engagement in State-level Activities, Partnerships (n=51). 


Awards. Although respondents were not as frequently engaged in state-level awards as the 
other types of activities described above, two-thirds reported engaging in these activities at some 
point in the previous 12 months. Forty percent of respondents (n=20) reported coordinating 
teacher awards, such as Presidential Awards for Excellence in Mathematics and Science 
Teaching (PAEMST), 28% (n=14) reported coordinating student scholarships, and 26% (n=13) 
reported conducting grant competitions often or all of the time (see Figure 3e). 


Coordinating teacher awards (e.g., PAEMST, Content Area 
Teacher of the Year) 


Coordinating student scholarships (e.g., NYSC, state 
: 24% 
scholarships) 


Conducting grant competition 


mNever Rarely Sometimes mOften maAll of the time 


Figure 3e. Frequency of Engagement in State-level Activities, Awards (n=51). 
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Teacher Professional Development 


We asked respondents to characterize the best opportunity in their states for teachers to learn 
about the Framework. Survey questions focused on eliciting attributes related to evidence- 
based practices reflected in recent consensus volumes (e.g., NASEM, 2015) and in the CSSS’ own 
Science Professional Learning Standards (Shaw et al., 2018). In relation to the best 
professional development (PD) offered in the last year, the majority (63% or n=34) of 
respondents reported that a state agency led the experience. Other frequent PD facilitators were 
district leaders and science teachers (reported by 39% and 32% of respondents, respectively). 
Twenty-seven percent of respondents also identified “other” PD leaders, including regional 
agencies (e.g., county offices of education), intermediary organizations, university faculty, or 
consultants. With regard to consultants, 24% of respondents indicated that an individual 
connected to CSSS led the PD, compared to 9% who indicated that someone not connected to 
CSSS led the PD. 


With respect to funding, the majority indicated that the PD was funded by one of two types of 
federal grants. A full 57% (n=33) indicated the PD was supported by a Teacher Quality 
Partnership grant, which are part of a program contained within Title II of the Higher Education 
Act designed to improve the preparation of prospective teachers and enhance professional 
development activities for new teachers. Another 15% of respondents (n=7) indicated that the 
PD was funded by a Supporting Effective Educator Development (SEED) grant, which was a 
program appropriated in the federal Every Student Succeeds Act (ESSA) that supports the 
implementation of evidence-based preparation, development, or enhancement opportunities for 
educators. The remaining respondents indicated that PD funding came from other sources, such 
as foundations or state-specific agencies. 


In terms of PD characteristics, we asked respondents about duration as well as the extent to 
which the PD afforded teachers with opportunities for collective participation, that is, learning 
with and from peers. !8 Related to duration, the majority of respondents (60% or n=32) 
indicated that the PD lasted between one and five days. Concerning collective participation, the 
majority of respondents (51%) reported that the PD afforded participants between 8 and 16 
hours (i.e., 1-2 days) of collective participation. 


Finally, we asked about the foci of PD that respondents felt offered teachers the best learning 
opportunities related to the Framework. In general, respondents indicated that this PD covered 
foundational concepts such as: (1) the three dimensions of science learning in the Framework or 
Next Generation Science Standards (NGSS), or (2) instructional planning aligned to the NGSS. 
They indicated that less attention was paid to three-dimensional assessments and developing 
students’ skill in asking questions or designing solutions. Specifically, a full 88% (n=49) of 
respondents indicated that the three dimensions of science learning in the Framework or Next 
Generation Science Standards (NGSS) were a major PD focus, whereas 31% indicated that 
three-dimensional formative or summative assessment was a major focus (see Figure 4). 
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Figure 4. Foci of Professional Development that Respondents Identified as Providing the Best Opportunity for 


Teachers to Learn about the Framework (n=55). 
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Research Use 


Key Findings 
e The research respondents reported using to support implementation of the 
Framework most often focused on student learning and classroom assessment, 
while the research they reported sharing with local leaders most often focused on 
classroom assessment and pedagogical practices. 


e The vast majority of named research did not focus exclusively on the needs or 
assets of particular student subgroups (e.g., by race/ethnicity, gender, 
socioeconomic status, or language). 


e Research reports or policy briefs, especially those published by the National 
Academies, as well as peer-reviewed journal articles, comprised the majority the 
research respondents used and shared. 


e The most commonly cited reasons that respondents felt a piece of research was 
trustworthy were if the findings applied to their state context, or if it gave them 
new ideas to support implementation of the Framework. 


e One-third of respondents indicated frequently accessing research through 
colleagues in their state department of education or through CSSS; few 
respondents indicated frequently accessing research through NSELA, a parallel 
professional organization for district leaders, or from the What Works 
Clearinghouse. 


e While the vast majority of respondents reported frequently looking for relevant 
research studies when confronted with a new problem, fewer reported contacting 
researchers to find out more about specific articles, especially researchers they did 
not already know. 


Research CSSS Members Use and Share 


We asked respondents to identify two pieces of research they use in their work. Research was 
defined as an activity in which people employ systematic, empirical methods to answer a specific 
question. First, we asked respondents to name a piece of research they used to inform their 
state’s decisions related to implementation of the Framework for K-12 Science Education. 
Second, we asked respondents to name a piece of research they shared with district or school 
leaders related to implementation of the Framework. Given that respondents who completed 
the survey at the 2017 CSSS annual meeting did not have Internet access, it is important to keep 
in mind that they may have been unable to locate specific information for each piece of research; 
thus, their responses may have been limited in unmeasurable ways. 
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Research used. With respect to specific pieces of research used to inform state 
implementation decisions, 17 unique pieces of research were listed by a total of 43 respondents, 
or 73% of the sample. The most frequently mentioned pieces of research were authored by the 
National Academies of Science, Engineering, and Medicine (NASEM) and included: A 
Framework for K-12 Science Education (2012), Guide to implementing the Next Generation 
Science Standards (2015), Developing Assessments for the Next Generation Science Standards 
(2014), and Taking Science to School (2007) (see Appendix B for a complete list). Respondents 
also named recently published journal articles, such as: “Climate confusion among U.S. 
teachers” by Plutzer, McCaffrey, Hannah, Rosenau, Berbeco, and Reid (2016), and “Building an 
assessment argument to design and use next generation science assessments in efficacy studies 
of curriculum interventions” by DeBarger, Penuel, Harris, and Kennedy (2016). 


Although we observed a higher response rate for this survey item in Year 2 than in Year 1 (2016, 
62%; 2017, 73%), as well as a higher total number of responses (2016, n=38; 2017, n=43), the 
number of unique pieces of research named was fewer: 17 in 2017 as opposed to 26 in 2016. Of 
the 17 unique pieces of research mentioned in Year 2, nine were also mentioned in Year 1 and 
most often included National Academies of Science, Engineering, and Medicine publications. 
Thus, even though the survey sample was more diverse in terms of CSSS membership in Year 2 
than in Year 1, there was more congruence across the pieces of research named to support state 
implementation of the Framework. 


Research shared. For the second item related to a specific piece of research shared with 
district or school leaders, 20 unique pieces of research were listed by a total of 37 respondents, 
or 63% of the sample. These included NASEM publications, such as Developing Assessments for 
the Next Generation Science Standards (2014) and Guide to Implementing the Next Generation 
Science Standards (2015) and a range of journal articles, like “Overview: How can we promote 
equity in science education?” by Bell and Bang (2015), as well as conference presentations and 
practice guides. As with the first item, even though there was a higher response rate in Year 2 
than in Year 1 (2016, 54%; 2017, 63%), and a higher number of responses (2016, n=33; 2017, 
N=37), there were fewer unique pieces of research mentioned: 20 in Year 2 compared to 24 in 
Year 1. Of the 20 pieces mentioned in Year 2, ten were also mentioned in Year 1. 


Below, we describe the focus of this research as well as the content areas and student subgroups 
the pieces of research highlighted. Then, we describe the form of the research referenced by 
respondents, the reasons it was useful, and with whom it was shared. 


Focus of Research 


Research used. The 17 unique pieces of research listed by respondents as useful in informing 
state decisions tended to focus on Teachers and teaching in the classroom (47% or n=8), 
followed by Student learning and student outcomes (29% or n=5), Assessment (29% or n=5), 
and School system organization, improvement and reform (6% or n=1) (see Figure 5a). 
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Figure 5a. Focal Topics of Unique Pieces of Research Used (n=17) and Shared (n=20). 


However, when looking at the pieces of research named across all respondents (i.e., rather than 
at unique pieces of research; see Figure 5b below), the majority (40% or n=17) focused on 
Student learning and student outcomes. All but one piece of research in this topic area focused 
on student learning and identity development, with the exception of Gonzali-Lee and colleagues’ 
(2015) report 2014-2015 STEM Pathways Evaluation, which focuses on student achievement 
and outcomes. 


The second most prominent category of research named by respondents was Assessment (26% 
or n=11 responses). Within this topic area, the main focus was classroom assessment (90% or 
n=10), with Developing Assessment for NGSS by NASEM (2014) most frequently mentioned, as 
well as a recent journal article, “Building an assessment argument to design and use next 
generation science assessments in efficacy studies of curriculum interventions,” by DeBarger et 
al. (2016) and published in the American Journal of Evaluation. Only one respondent 
referenced a piece of research focused on standardized testing. 


The third most prominent category of research named by respondents was Teachers and 
teaching in the classroom (21% or n=9). Within this topic area, 55% (n=5) focused on 
pedagogical practices, such as the 2015 book, A Vision and Plan for Science Teaching and 
Learning, by Brett Moulding (a CSSS honorary member), Rodger Bybee, and Nicole Paulson (a 
CSSS associate member). Then, 22% (n=2) dealt with teacher effectiveness and evaluation, 11% 
(n=1) with teacher professional learning, and 11% (n=1) with other subtopics. 


The least frequent research topic named by respondents was School system organization, 
improvement and reform with 14% (n=6) focusing on system improvement, and all referencing 
the National Academies’ 2015 Guide to Implementing the NGSS. 
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Figure 5b. Focal Topics of Pieces of Research Used (n=43) and Shared (n=37) Across All Respondents. 


Research shared with others. Of the 20 unique pieces of identifiable research listed by 
respondents as shared with district or school leaders, 45% (n=9) focused on Teachers and 
teaching in the classroom, 25% (n=5) on Student learning and student outcome, 15% (n=3) on 
Assessment, and 15% (n=3) on School system organization, improvement and reform (see 
Figure 5a). However, when analyzing the research pieces named across all respondents (i.e., 
rather than unique pieces of research; see Figure 5b), the most prominent category named was 
Assessment, with approximately one-third of respondents (31% or n=11) stating that they shared 
a piece of research related to this topic. Nearly all of these pieces pertained to classroom 
assessment (91% or n=10) and referenced Developing Assessments for the Next Generation 
Science Standards (NASEM, 2014). 


The second most frequent type of research shared by respondents focused on Teachers and 
teaching in the classroom, with 29% (n=10) of respondents identifying a piece of research on 
this topic. Half of these pieces focused on pedagogical practices, for example the research brief 
Ready, Set, Science (Michaels et al., 2008). The other half was distributed across the following 
subtopics: teacher professional learning (30% or n=3), teacher effectiveness and evaluation 
(10% or n=1), and curriculum (10% or n=1). 


The third category of research shared was Student learning and student outcomes, with 23% of 
respondents (n=8) naming a piece of research related to this topic. The majority (88% or n=7) 
focused on student learning and identity development, for example citing “Overview: How can 
we promote equity in science education?” (Bell & Bang, 2015), which highlights the importance 
of connecting students’ interests and experiences to support sense-making in science. The 
remaining piece of research focused on student achievement and learning outcomes. 
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With respect to School system organization, improvement and reform, 17% of respondents 

(n=6) indicated that they shared research related to this topic. A majority of these pieces focused 
on system improvement (83% or n=5), with most mentioning the Guide to Implementing the 
NGSS (NASEM, 2015). The other piece mentioned was Tate’s (2001) article “Science education 
as a civil right: Urban schools and opportunity-to-learn considerations,” which argues that the 
provision of high-quality science education is a civil rights issue, particularly in segregated 
urban schools. 


Subject matter and subgroups. The overwhelming majority (88% or n=15) of the research 
pieces used by respondents to inform their state’s implementation of the Framework focused on 
science and engineering. Only two pieces focused on other subjects: one on literacy and one on 
multiple content areas. This distribution was similar for pieces of research respondents 
indicated sharing with local leaders, with 80% (n=16) focused on science and engineering, 15% 
(n=3) on general topics, and 5% on literacy (n=1). 


Subgroups of students. The vast majority of research studies respondents named did not 
focus on a particular subgroup primarily or exclusively (n=16 or 94% of research used, and n=17 
or 85% of research shared). Of the research used to inform state’s decisions, one piece focused 
on several groups (i.e., SES/poverty, race/ethnicity), while three pieces of research shared with 
local leaders focused on particular subgroups, with one discussing multiple groups, one focused 
on English learners, and one examining race and ethnicity. The NASEM consensus volumes that 
were often named by respondents focus on student subgroups, but not exclusively. 


Forms of research. The pieces of research that respondents used to inform their state’s 
implementation decisions included research reports or policy briefs (n=7 or 41%), peer-reviewed 
journal articles (n=5 or 29%), practitioner-oriented research pieces (n=2 or 12%), and one each 
of the following: book, research-based tool, and technical report (see Figure 6 below). 


The pieces of research respondents mentioned sharing with district or school leaders included 
research reports or policy briefs (n=7 or 35%), peer-reviewed journal articles (n=3 or 15%), 
practitioner-oriented research pieces (n=3 or 15%), books (n=2 or 10%), research-based tools 
(n=2 or 10%), conference presentations (n=2 or 10%), and standards documents (n=2 or 5%). 
Overall, the forms of research used and shared in Year 2 were more diverse than in Year 1. 
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Figure 6. Forms of Research Used (n=17) and Shared (n=20). 


Reasons why a piece of research was useful. For the item in which respondents listed a 
piece of research they used to inform their state’s decisions related to implementation of the 
Framework, we also asked “Why was it useful?” Responses from 34 respondents provided 
enough information to identify reasons. An additional five respondents provided no response, 
and another four did not provide specific reasons; these nine responses were excluded from the 
analysis. 


Respondents tended to report using a particular piece of research to support their own 
professional learning (n=11 or 32%), design programs or initiatives (n=9 or 27%), and persuade 
others about the value of programs or initiatives (n=6 or 18%). Fewer respondents indicated 
that the research was useful for providing instructional leadership for others in a central office 
or at school sites (n=2 or 6%), or evaluating programs, policies, initiatives (n=2 or 6%). 


Groups with whom research was shared. For the item in which respondents listed a piece 
of research they shared with local leaders, we also asked “With whom did you share it?” Thirty 
respondents provided enough information to identify with whom they shared research; five 
respondents did not provide an answer. Overall, respondents reported sharing research with 
individuals at a variety of levels, including other state leaders (n=6 or 20%), district leaders (n=5 
or 17%), teacher leaders or teachers (n=5 or 17%). And, nearly half of the respondents reported 
sharing research with more than one level (n=14 or 47%). 
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Research trustworthiness. Among the 43 respondents that mentioned a piece of research 
used to inform state decisions about implementation of the Framework, 95% (n=41) responded 
to the item asking what made that research trustworthy. The three most commonly cited 
reasons were: (1) they could relate the research findings to their state context (78% or n=32), (2) 
it gave them new ideas to support implementation of the vision of the Framework (71% or 
n=29), and (3) someone they trusted conducted the research (see Figure 7). A less commonly 
cited reason was if the research methods were rigorous (36% or n=15). 


| could relate the findings to my state context ee 78% 
It gave me new ideas for how to support implementation of 
71% 
the Framework 


Someone | trust conducted the research 


It deepened my understanding of the vision of science eeu 
learning in the Framework ° 
Someone | trust shared the piece of research with me EC 46% 
The research methods used are rigorous cE 36% 


My state commissioned the research [J 5% 


| was involved in the research CI 5% 


Figure 7. Reasons Why a Piece of Research Was Trustworthy (n=41). 


Sources CSSS Members Used to Access Research 


A set of survey questions asked respondents to report how often they sought out or acquired 
research from various sources in the last year (see Figure 8). Respondents to this question 
(n=58) tended to access research most frequently through formal and informal networks, 
mainly through CSSS (60% often or all the time, n=35), university researchers (60% often or all 
the time, n=35), other colleagues in state departments of education (58% often or all the time, 
n=32), or the National Science Teacher Association (50% often or all the time, n=29). 
Additionally, research conferences and state consultants were mentioned as prevalent sources of 
research, with 50% and 23% of respondents reporting seeking out or acquiring research from 
these sources often or all the time, respectively. Respondents were less likely to report accessing 
research at that frequency from the National Science Education Leadership Associations 
(NSELA), vendors (5%, n=3), or the What Works Clearinghouse (7%, n=4). These findings are 
comparable with Year 1 results. 
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Figure 8. Sources CSSS Members Used to Access Research (n=58). 


CSSS Members’ Efforts to Acquire Research 


We asked respondents five questions related to their efforts to acquire research and to develop 
relationships with researchers when confronted with new problems or decisions. The vast 
majority of respondents (n=48 or 83%) reported looking for relevant research studies often or 
all of the time when confronted with a new problem or decision (see Figure 9). Additionally, they 
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overwhelmingly indicated (n=46 or 74%) finding it valuable to consult educational research 
often or all the time when confronted with a new problem or decision. Although 42% of 
respondents (n=24) reported that they contact researchers they already know often or all of the 
time, few reported reaching out to researchers to find out more about specific articles at that 
frequency (n=9 or 15%) or contacting researchers they did not know (n=3 or 5%). These results 
were similar between Year 1 and Year 2. 


When confronted with a new problem, | look for research nee, 
studies that might be relevant i 
When confronted with a new problem or decision, | find it 
: 2% 3% 21% 
valuable to consult education research 
When confronted with a new problem or decision, | contact ne 
researchers | already know who have relevant expertise ‘ 
| contact researchers to find out more about articles they Aen 
have written 
When confronted with a new problem or decision, | contact 
researchers | don't already know who have relevant 35% 
expertise 


m Never Rarely Sometimes mOften mAll the time 


Figure 9. Effort to Acquire Research (n=59). 
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Research Networks 


Key Findings 
e Associate, honorary, and affiliate CSSS members served as sources of research to 
support state implementation of the Framework in the areas of curriculum, 
instruction, assessment, and professional development. University faculty not 
affiliated with CSSS were also prominent sources of research. 


e NewCsSSS state members were more likely than veteran CSSS members to name 
individuals from their own state as sources of research. 


e While some CSSS members facilitated the exchange of research between members 
in different states, others facilitated the exchange of research between CSSS 
members and researchers unaffiliated with the association. 


e Formal CSSS activities, particularly substantive meetings, were important for 
facilitating the exchange of research related to the Framework. 


CSSS Members’ Research Networks 


We asked respondents to name individuals to whom they turned in the past year for research to 
inform their state’s efforts to implement the vision of the Framework for K-12 Science 
Education. The findings reported below focus on those interactions that occurred at least 3-4 
times per year, in an effort to capture research exchanges that took place outside of the annual 
CSSS meeting. Although the meeting likely provided state members structured opportunities to 
share information and ideas, and to come into contact with researchers, we were most interested 
in the informal interactions occurring outside of this formal networking opportunity. In this 
section we analyze three aspects of respondents’ research-related interactions: (a) how many 
individuals they turned to for research, and how these relationships were affected by 
membership type, experience, and research area; (b) what the characteristics of prominent 
sources and brokers of research were; and (c) whether and how CSSS activities and network 
characteristics were related. 


Research interactions by experience and membership type. We compared 
respondents’ research-related interactions by experience and member type (i.e., state, associate, 
honorary, and affiliate). To examine differences by experience, we compared results for new 
state members to more veteran state members, with new members defined as individuals who 
have been state members for less than two years. 


First, we examined how many people each respondent reported turning to for research, a 
measure called out-degree centrality. Table 2 displays summary statistics for this measure for 
each research topic (i.e., instruction, curriculum, assessment, and professional development), by 
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member type and experience. These results show that, for all topics, respondents turned to an 
average of one or two individuals for research. Nonetheless, there were notable differences in 
out-degree between new and veteran state members. Whereas new state members turned to an 
average of one individual for research related to instruction, with a maximum of seven, veteran 
state members turned to an average of two individuals, with a maximum of eleven. Differences 
were similar for curriculum and assessment, suggesting that veteran members sought out more 
individuals for research than new members in these areas. However, both new and veteran state 
members reported turning to an average of two individuals, and a maximum of nine, for 
research related to professional development. 


Table 2. Average Number of Individuals Respondents Turned to for Research. 


Mean sD Minimum Maximum 

New state members (n=15) 

Instruction 1.4 2.2 0 7 

Curriculum 1.1 1.4 0 4 

Assessment 1.1 1.6 0 5 

Professional development 1.7 2.6 0 9 
Veteran state members (n=26) 

Instruction 2.2 3.2 0 11 

Curriculum 1.8 2.6 0 10 

Assessment 1.8 2.4 0 7 

Professional development 2.2 2.6 0 9 
Honorary and associate members (n=11) 

Instruction 1.3 1.6 0 5 

Curriculum 0.9 0.9 0 3 

Assessment 2.1 3.4 0 11 

Professional development 2.1 2.7 0 9 
Affiliate members (n=6) 

Instruction 1.8 2.9 0 6 

Curriculum 1.3 1.5 0 4 

Assessment 1.5 2.1 0 5 

Professional development 1.8 3.0 0 7 


Second, we categorized the individuals to whom respondents turned for research into six 
groups: (1) CSSS state member, (2) associate or honorary member, (3) affiliate member, (4) 
unaffiliated university faculty, (5) intermediary (i.e., a researcher or professional development 
provider working for an intermediary organization), and (6) state or local leader. Figures 10a-d 
show the representation of these groups for each research topic and member type. 
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Results displayed in Figure 10a show that university faculty who were not affiliate members 
(referred to as “unaffiliated university faculty”) were most often named as sources of research 
related to instruction for new and veteran state members, as well as affiliate members 
(representing 48%, 38%, and 55% of individuals they sought for instruction-related, 
respectively). On the other hand, associate and honorary members most often named CSSS 
affiliates. Further, whereas state members and associate and honorary members named other 
associate or honorary members as individuals from whom they sought research related to 
instruction, none of the affiliate members named CSSS associate or honorary members. All 
types of members, however, indicated turning to state members and intermediaries for research 
related to instruction. 


60% 
50% 
40% 
30% 


20% 


~ ee i. I i | 


New State Member (n=15) Veteran State Member Associate or Honorary Affiliate Member (n=6) 
(n=26) Member (n=11) 


m State member m Associate or honorary member ww Affiliate member 


Unaffiliated university faculty mIntermediary mw State or local leader 


Figure 10a. Composition of Research Interactions Focused on Instruction. 


Turning to the individuals that respondents named as sources of research related to curriculum 
(see Figure 10b), we similarly observed that unaffiliated (i.e., non-member) university faculty 
were important sources of research in this area for new and veteran state members, as well as 
for associate and honorary members. Affiliate members, however, were most likely to name 
other affiliates. Then, while all member types named state members as curriculum-related 
sources of research, only new and veteran members named associate or honorary members as 
individuals they turned to for research related to curriculum. As with research related to 
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instruction, only veteran members named state or local leaders as individuals they turned to for 
research pertaining to curriculum. 
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50% 
40% 


30% 


20% | | 
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New State Member (n=15) Veteran State Member Associate or Honorary Affiliate Member (n=6) 
(n=26) Member (n=11) 


m State member m Associate or honorary member m Affiliate member 


m Unaffiliated university faculty m Intermediary @ State or local leader 


Figure 10b. Composition of Research Interactions Focused on Curriculum 


Results for research interactions related to assessment differed, with unaffiliated university 
faculty named less often, and associate, honorary, and affiliate members serving as key sources 
of assessment-related research (see Figure 10c). For example, associate and honorary members 
represented 29% of the individuals to whom new state members turned for research related to 
assessment, compared to 3% of the individuals that new members named for research related to 
curriculum. The exception was among affiliate members, who did not name any associate or 
honorary members. As with instruction and curriculum, state members were named by all 
member types as sources of research related to assessment. Finally, similar to results for 
curriculum, affiliate members did not name any intermediaries as sources of research related to 
assessment, although all other types of members did. 
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Figure 10c. Composition of Research Interactions Focused on Assessment 


With respect to the individuals named as sources of research related to professional 
development, unaffiliated university faculty were again the most often named by all member 
types (see Figure 10d). Affiliate members also remained prominent, as did state, associate, and 
honorary members. The exception was for affiliate members, who did not name any associate or 
honorary members as individuals they turned to for professional development-related research; 
however, intermediaries represented a large proportion of their sources of research in this area. 
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Figure 10d. Composition of Research Interactions Focused on Professional Development 


Third, we examined the extent to which respondents named individuals within their own states 
as sources of research related to each topic area (see Table 3). Our results showed that new state 
members were twice as likely than veteran members to name individuals from their own state as 
sources of research related to instruction, curriculum, assessment, and professional 
development. Additionally, associate and honorary members often named individuals within 
their own state for research related to curriculum. Finally, affiliate members rarely named 
individuals from their own states as sources of research in any area. 


Table 3. Proportion of Research Interactions that were Within State. 


New State Veteran State Associate or Affiliate 
Members Members Honorary Members 
(n=15) (n=26) Members (n=11) (n=6) 
Instruction 57% 32% 43% 9% 
Curriculum 50 35 60 13 
Assessment 47 28 36 11 
Professional development 64 31 43 9 
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Prominent sources and brokers of research. To understand the kinds of individuals who 
might have the most influence on the research ideas exchanged among CSSS members, we 
identified the most prominent sources and brokers of research in the network. To identify 
prominent sources, we calculated in-degree centrality, or the number of individuals who named 
each person in the network, for each research topic. We considered anyone who was named by 
more than two people as a prominent source of research in that network. Across all research 
areas (i.e., instruction, curriculum, assessment, and professional development), nine individuals 
emerged as particularly influential; they are described in Table 4 below. Six of those nine 
individuals were the same across all four areas and include: two affiliate members, one honorary 
member, one associate member, one state member, and one unaffiliated university faculty 
member. 


Table 4. Individuals Most Frequently Named as Sources of Research. 


Number of Individuals Who Named Each Person 


Member Organization : ; Professional 
No. Instruction Curriculum Assessment 
Type Type Development 
1 Affiliate University 9 6 12 11 
2 Affiliate University 9 6 10 11 
3 Honorary Intermediary 5 8 6 10 
4 Associate Intermediary 4 3 4 4 
5 State Department of 
; 3 4 3 5 
Education 
6 Unaffiliated University 4 4 3 4 
7 Associate Intermediary 0 0 6 5 
8 State Department of 
} 3 0 0 0 
Education 
9 Unaffiliated University 0 3 0 0 


To explore how brokers might influence the kinds of research shared in the network, we 
calculated a measure called betweenness centrality. This measure allowed us to identify those 
individuals who facilitated the exchange of research between two individuals who would not 
otherwise be connected. As an example, person B in Figure 11 below serves as a broker between 
person A and C, as person A goes to B for research, and B goes to C for research, but A and C are 
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not connected. On the other hand, person B is not a broker between person C and person D, as 
they are already connected. 


G) 
O~9 


Figure 11. Example of a Broker. 


Given the nature of brokering, only individuals who responded to the survey can be identified as 
research brokers. Across all four research areas, three individuals emerged as key brokers: one 
state member, one associate member, and one affiliate member, and each played different kinds 
of brokering roles within the association and among states. For instance, the state member 
tended to broker research between unaffiliated university faculty within their state and state and 
affiliate members in other states. In this way, they facilitated CSSS members’ access to research 
conducted by unaffiliated academics in their state. The associate member, in contrast, facilitated 
state members’ access to research used by associate, honorary, and affiliate members in 
different states. The affiliate member also facilitated state members’ access to research from 
other CSSS affiliates, but they also brokered connections between members and unaffiliated 
university faculty. These exchanges occurred between individuals from within and outside of the 
affiliate member’s state. 
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Relating Association Activities and Research Networks 


As a final step in our analysis, we examined relationships between respondents’ reported 
engagement in CSSS activities (see Figure 2) and the extent to which they were sought out for 
research related to the Framework or served as a broker of such research. Although the small 
sample size (n=42) limited the extent to which these relationships could reach significance, we 
found that respondents’ participation in formal CSSS meetings were positively and significantly 
associated with both the number of individuals who named them as sources of research (i.e., in- 
degree centrality), and the extent to which they served as a broker in the research network (i.e., 
betweenness centrality). 


Specifically, respondents who indicated more frequent participation in CSSS annual meetings 
over the previous three years were significantly more likely to be sought out for research related 
to the Framework, and these relationships were significant across all research topics (i.e., 
curriculum (1r(41)=0.42, p<.01), instruction (r(41)=0.36, p<.05), assessment (r(41)=0.39, 
p<.05), and professional development (r(41)=0.37, p<.01)). 


Further, respondents who reported more frequent participation in BCSSE meetings over the 
previous three years were significantly more likely to be sought out for research, and to serve as 
research brokers. These relationships were significant across all research topics for the former 
(curriculum (r(41)=0.54, p<.001), instruction (1(41)=0.45, p<.01), assessment (r(41)=0.50, 
p<.01), and professional development (r(41)=0.49, p<.o1)), and for two of four topics for the 
latter (curriculum (r(41)=0.37, p<.05), instruction (r(41)=0.31, p<.10), assessment (r(41)=0.41, 
p<.05), and professional development (r(41)=0.21, p<.21)). These findings suggest that formal 
CSSS activities, and particularly substantive meetings, rather than organizational meetings (i.e., 
committee meetings) and more informal activities (e.g., webinars), are important for facilitating 
the exchange of research related to the Framework among CSSS members. 
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Key Conclusions 


Findings from this study suggest that CSSS members were highly engaged in state decisions 
related to science standards and assessments, but were less involved with the implementation of 
curricula or in decisions related to resource allocation or contract selection. They were also 
highly involved in the design and implementation of professional development, the best of 
which tended to focus on foundational concepts in the Framework for K-12 Science Education. 


With respect to the kinds of research that CSSS members found useful and shared with others, 
consensus reports by the National Academies of Science, Engineering, and Medicine were 
particularly prominent. Although these reports often touch on the needs and assets of specific 
students (e.g., by race/ethnicity, socioeconomic status, gender, or language), very few of the 
research pieces that CSSS members used or shared focused exclusively on fostering equitable 
science learning opportunities for particular subgroups. In the context of changing 
demographics, it will be important to consider whether and how science education reforms are 
implemented with consideration for students across race, gender, socioeconomics, and 
language, and for research to address these issues. 


Furthermore, given that school system organization and reform was the least frequently 
mentioned topic of research, it will be important for members to consider scholarship on 
organizational change and the development of infrastructures to foster coherence across levels 
of the system. Examining research on both equity and school system reform is especially 
important given CSSS’ current emphasis on building coherent and equitable systems of science 
education via the ACESSE project. The recent NASEM report (2018) focused on transforming 
teaching and learning for English learners (ELs) in STEM may be useful in addressing these 
research areas, as well as the broader scholarship on designing systems with coherence and 
equity in mind.9 


CSSS members reported regularly using research to inform their state’s decisions related to 
implementation of the Framework. They tended to access research through their networks, such 
as from CSSS or state department of education colleagues, rather than via other research outlets 
such as governmental websites. Moreover, broad engagement in CSSS networks, especially 
through substantive meetings, appeared to facilitate the use of research more than activities like 
planning meetings or webinars. Findings also showed that the CSSS membership structure was 
important for facilitating the exchange of research related to statewide implementation of the 
Framework, as associate, honorary, and affiliate members were all key sources of research. 


In conclusion, state science leaders tended to access and make use of relevant research to inform 
state-level efforts to implement the Framework, and had a robust and well-connected research 
network. At the same time, this study identified potential areas of growth to amplify the 
strategic place these leaders hold in the process of providing a quality science education to an 
increasingly diverse population of students. 
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Appendices 


Appendix A. Survey Instrument 


Survey: Research Use Among State Science Leaders 


To begin, please provide the following information. While we need this information to track 
responses over time, please remember that you will be assigned a study ID number, and will 
never be identified by name in the dataset or in any data reports. 


1. First name: 


2. Last name: 


3. Job title: 


4. Organizational affiliation (if applicable): 


5. State: 


6. Lama: [MARK ALL THAT APPLY] 
Ll Current CSSS member 
L] Former CSSS member 
Ll CSSS affiliate member 
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Definition of Research: For the purposes of this study, “research” is an activity in which people employ 
systematic, empirical methods to answer a specific question. Research, as we define it, is different than 
the practice of looking at data from the district, school, or classroom, which is more open-ended and 
seldom addresses specific research questions. 


The next set of questions ask about your use of research related to implementation of the Framework 
for K-12 Science Education. 


7. First, think about a time when you used a piece of research to inform a decision in your state 
related to the implementation of the Framework for K-12 Science Education. What was that piece 


of research? While you do not need to enter all details, please provide as much information as 
you can about this piece of research so we can locate it ourselves. 


Title: 


Author: 


Year Published: 


Publisher: 


Topic: 


Why was this piece of research useful? 


What made this piece of research trustworthy? (MARK ALL THAT APPLY.) 

I was involved in the research. 

My state commissioned the research. 

Someone I trust conducted the research. 

Someone I trust shared the piece of research with me. 

The research methods used are rigorous. 

I could relate the findings to my state context. 

It deepened my understanding of the vision of science learning in the Framework. 
It gave me new ideas for how to support implementation of the Framework. 
Other: 


Wo oe ooo oo 


Findings from a Survey of State Science Leaders: Year Two 43 


National Center for Research in Policy and Practice 


Definition of Research: For the purposes of this study, “research” is an activity in which people employ 
systematic, empirical methods to answer a specific question. Research, as we define it, is different than 
the practice of looking at data from the district, school, or classroom, which is more open-ended and 
seldom addresses specific research questions. 


8. Now, think about a time when you shared a piece of research with district or school 
leaders related to the implementation of the Framework for K-12 Science Education. What 


was that piece of research? While you do not need to enter all details, please provide as 
much information as you can about this piece of research so we can locate it ourselves. 


Title: 


Author: 


Year Published: 


Publisher: 


Topic: 


With whom did you share this piece of research? 


Why did you share this piece of research? 


Findings from a Survey of State Science Leaders: Year Two 44 


National Center for Research in Policy and Practice 


Definition of Research: For the purposes of this study, “research” is an activity in which people employ 
systematic, empirical methods to answer a specific question. Research, as we define it, is different than 
the practice of looking at data from the district, school, or classroom, which is more open-ended and 
seldom addresses specific research questions. 


9. How often do you do each of the following? 
All of 


Never Rarely Sometimes Often fhe aime 


A. I contact researchers to find 
out more about articles they fe) oO oO oO oO 
have written. 


B. When confronted with a new 
problem or decision, I 
contact researchers I already oO Ol oO oO oO 
know who have relevant 
expertise. 
C. When confronted with a new 
problem or decision, I 
contact researchers I don't O OC CO im O 
already know who have 
relevant expertise. 


D. When confronted with a 
new problem or decision, I 


look for research studies U U U U O 
that might be relevant. 
E. When confronted with a 
new problem or decision, I 
find it valuable to consult O O O O O 
education research. 
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Definition of Research: For the purposes of this study, “research” is an activity in which people employ systematic, empirical methods to answer 
a specific question. Research, as we define it, is different than the practice of looking at data from the district, school, or classroom, which is more 
open-ended and seldom addresses specific research questions. 


10. In the past 12 months, to whom have you turned for research that can inform your state’s efforts to implement the vision of the Framework for K-12 
Science Education? Please include up to 10 researchers, state science supervisors, or other colleagues, within or outside your state. For each person 
listed, please indicate their organization, the frequency with which you interacted with them, and the general topics of research discussed. You do not 
need to fill in all the spaces. While we ask you to include individual names, all individuals will be assigned an ID number and will ever identified by 
name in datasets or data reports. 


5; ; D 
How often do you turn to each person for research? Wiha) Obie. cbuese arch de you discuss) 


MARK ALL THAT APPLY. 

1-2 3-4 Every 2 2-3 times Bastiesional 

First and Last Name Organization times times months | Monthly per Weekly | Curriculum | Assessment aa pel dieisiecta’ 
per year | per year or so month ‘ile! as acer 
l. L] L L L L L L L L im 
2. L L L L L L L L L 
3: L L L L L L L L L C] 
4. L L L L L L L L L CL] 
5. L L L L L L L L L C] 
6 L L L L L L L L L | 
7. L L L L L L L L L L] 
8. L L] L L L L L L L CL] 
9. L L L L L L L L L CL] 
10. L L L] L L L L L L] CL] 
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Definition of Research: For the purposes of this study, “research” is an activity in which people employ 
systematic, empirical methods to answer a specific question. Research, as we define it, is different than 
the practice of looking at data from the district, school, or classroom, which is more open-ended and 
seldom addresses specific research questions. 


11. During the past 12 months, how often have you sought out or acquired research from the 
following sources? 


Never Rarely | Sometimes | Often at We te 
time 

A. National Science Teachers 

Association (NSTA) r O O O O 
B. National Science 

Education Leadership oO oO oO oO oO 

Association (NSELA) 
C. Council of State Science 

Supervisors (CSSS) O ri r O O 
D. University researchers Oo Oo Oo Oo oO 
E. Regional Education 

Laboratories (RELs) 0 0 r r O 
F. County offices of 

education UI O a O O 
G. Other colleagues in the 

state department(s) of oO oO oO oO oO 

education 
H. National Center for 

Education Statistics Oo Oo oO oO Oo 

(NCES) 
I. What Works 

Clearinghouse (WWC) 0 0 0 ry O 
J. Newspaper or magazine 
K. Vendors 
L. Consultants working with 

the state O O O O O 
M. Conferences where 

research is presented r a O O O 
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eee 
The next set of questions asks about your work as a state science leader. 


12. In the last 12 months, how often have you engaged in the following activities as part of your 
role as a state science leader? 


Never Rarely Sometimes Often “a oF oe 
time 

Standards & Curriculum 
A. Reviewing or developing state 

science standards (in lead role) O O rm r rl 
B. Reviewing or developing state 

science standards (in support role) O O rm r rT 
C. Organizing state curriculum 

adoption 0 O O O O 
D. Consulting with curriculum 

companies on products in the design O oO O O O 

phase 
E. Advising state committee’s 

decisions on science course content 

and/or graduation policies (in lead r r 0 0 O 

role) 
F. Advising state committee’s 

decisions on science course content 

and/or graduation policies (in O O O O O 

support role) 
Assessments 
G. Designing state assessments (in lead 

aie Oo Oo Oo Oo Oo 
H. Designing state assessments (in 

support role) 0 0 0 0 O 
I. Selecting contractors for state 

assessments r = 0 0 O 
Professional Development 
J. Allocating Title 2A funds im im oO oO oO 
K. Identifying vendors to provide 

professional development to support LJ LJ L] L LC] 

state education initiatives 
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All of the 


Never Rarely Sometimes Often ; 
time 


L. Writing contracts for professional 
development providers to support LJ L] L] L CL] 
state education initiatives 


M. Designing professional development 


(in lead role) O U O U U 
N. Designing professional development 

(in support role) O O O O O 
O. Conducting professional 

development O O O O O 
Partnerships 
P. Identifying resources to share with 

districts 0 O O O O 
Q. Screening or reviewing resources to 

share with districts O O O O O 
R. Collaborating with your state 

eres Oo Oo Oo Oo Oo 


university system in K-16 initiatives 


S. Establishing partnerships with 
business, industry, and non-formal oO oO Ol im Ol 
education groups 


Awards 


T. Coordinating student scholarships 
(e.g., NYSC, state scholarships) L L L LI C 


U. Coordinating teacher awards (e.g., 


PAEMST, Content Area Teacher of O oO oO CO Ol 
the Year) 
V. Conducting grant competition O ia oO oO Oo 
Other 
W. Other — Please list: 
LJ | L] LJ | 
X. Other — Please list: 
LJ LJ L] L] | 
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The next group of questions ask about professional development (PD) offered in your state. 
When answering these questions, please think about the PD your state has offered in the last 
year that you think offered the best opportunity for teachers to learn about the Framework for 
K-12 Science Education. 


13. Who led this professional development? (MARK ALL THAT APPLY) 


CL] A state agency leader 
C1 A district leader 

L] A science teacher 

L) A commercial vendor from within the state 

L] A commercial vendor from outside the state 

L) An individual consultant who is connected to CSSS 

L] An individual consultant who has no connection to CSSS 
L] Other — Please specify: 


14. How much time did participants spend together in professional development? 


<1 hour 

1-2 hours 

3-5 hours 

6-8 hours 

9-16 hours 

17-24 hours 

25-40 hours 

41-80 hours 

More than 80 hours 


OOOOO00O00 


15. Over what period of time did the professional development occur? 


Less than a day 

One day 

Two to four days 

A week 

A month 

Two to five months 
Six to nine months 
Ten to twelve months 
More than a year 


OOOOO00O00 


Findings from a Survey of State Science Leaders: Year Two 50 


National Center for Research in Policy and Practice 


16. Which topics did the professional development address? 


A major A minor Nota 
focus focus focus 
A. The three dimensions of science learning in 
The Framework or NGSS 0 0 0 
B. Designing materials aligned to the NGSS or 
new state standards UO O 
C. Planning instruction aligned to the NGSS or 
new state standards 0 0 0 
D. How to develop students’ skill in asking 
questions ry ry O 
E. How to develop students’ skill in 0 q 0 
constructing and using scientific models 
F. How to develop students’ skill in designing 
solutions to engineering problems 0 0 0 
G. How to develop students’ skill in engaging 
in argument from evidence O UO H 
H. How to design three-dimensional formative 
Oo Oo Oo 


or summative assessment items 


17. Below are several funding streams for professional development. Which of these streams will 
your state use to fund professional development in science in the future, either the group 
grants to schools and districts or through state programs? (MARK ALL THAT APPLY) 


LI Title If — Teacher Quality Partnership Grants 

L] Title Il — Teacher Incentive Grants 

L] Title II — Supporting Effective Educator Development (SEED) Grants 
CL] STEM Master Teacher Corps 

CL] School Improvement Grants 

O 


Other: 
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ee 
The next questions ask about your engagement with the Council of State Science Supervisors. 


18. What roles have you taken on as a member (current, former, or affiliate) of the Council of 
State Science Supervisors? (MARK ALL THAT APPLY.) 


O 


President 
Secretary 


Board member 
Ad-Hoc Committee Chair 
Ad-Hoc Committee Contributor/Participant 


Presenter at this year’s CSSS conference 
Presenter at a previous CSSS conference 
Participant in state science conference 
Organizer of state science fair 

Other: 


None of the above 


EE ee oe IE). ee 
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Definition of Research: For the purposes of this study, “research” is an activity in which people employ systematic, empirical methods to answer 
a specific question. Research, as we define it, is different than the practice of looking at data from the district, school, or classroom, which is more 
open-ended and seldom addresses specific research questions. 


19. Please indicate which CSSS activities you have participated in over the last three years. For those activities in which you participated, please 
indicate the frequency as well as how each of the following activities supported your ongoing work. 


In the last three years, about how often have If you participated in the activity, in what ways did it 
you participated in each activity? support your work? (MARK ALL THAT APPLY.) 
Oren Ilearned about | I learned about I learned about 
Never Once 2-3 times : research research-based | strategies for addressing 
times ; : : 
findings. tools. issues in my state. 

A. CSSS Annual Meeting O O O O O EJ EJ 
B. CSSS Committee Meetings oO oO O O O O O 
C. CSSS Board Meetings O O O O O LC] LC 
D. BCSSE Meetings O O O O O L] O 
E. Collaborating with other states O L L im CL LC] L] 
F. Visiting other states O O O L L O L 
G. Consulting with CSSS members O O O O O LC] LJ 
H. Presenting at NRC or other 

national meetings 0 0 O O OH a UW 
I. Attending workshops/talks by 

researchers 0 O O OH LO L M 
J. Participating in CSSS sponsored 

webinars O O O O O a UW 
K. Reading information from the 

CSSS listserv 0 O O O a a UW 
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The final set of questions ask for demographic and background information. Your responses will 
not be used in identifiable ways, but will help us understand patterns across survey participants. 


20. Please indicate your race/ethnicity: 

White 

Black or African American 

Latino/Latina or Hispanic 

American Indian or Alaska Native 

Asian 

Native Hawaiian or Other Pacific Islander 
Two or more races 

Other 


ca ea 


21. Please indicate your sex: 
L] Male 
L] Female 


22. Including this year, for how many years have you served (or did you serve) as a state science 
supervisor? 


23. Including this year, for how many years have you served as an affiliate member of CSSS? 


24. Which degrees have you acquired? (MARK ALL THAT APPLY) 
Associate’s degree 

Bachelor’s degree 

Master’s degree 

Doctoral degree 

None of the above 

Other: 


a fo 


25. Which certifications do you hold or have you held in the past? (MARK ALL THAT APPLY) 
Teaching certification (general) 

Teaching certification (science) 

Administrator certification 

None of the above 

Other: 


SS) ee 


END OF SURVEY 
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Appendix B. Complete List of Pieces of Research Named by Respondents, in Alphabetical Order 


Citation No. of 
Respondents 
Who Used It 


Banilower, E. R., Smith, P. S., Weiss, |. R., Malzahn, K. A., Campbell, K. M., & Weis, A. M. (2013). Report of the 2012 
National Survey of Science and Mathematics Education. Chapel Hill, NC: Horizon Research, Inc. 


Bell, P. & Bang, M. (2015). Overview: How can we promote equity in science education? STEM Teaching Tools Initiative, 
Institute for Science + Math Education. Seattle, WA: University of Washington. Retrieved from 
http://stemteachingtools.org/brief/15 


Council of State Science Supervisors. (2017). Science Professional Learning Standards. Professional Learning Committee: 
Sam Shaw, Brett Moulding, Shari Templeton, Catherine Mackey, William Penuel, Katie Van Horne. Council of State Science 
Supervisors. 


DeBarger, A., Penuel, W., Harris, C., & Kennedy, C. (2016). Building an Assessment Argument to Design and Use Next 
Generation Science Assessments in Efficacy Studies of Curriculum Interventions. American Journal of Evaluation, 37(2), 
174-192. 


Gozali-Lee, E., Mueller, D., Streich, F., & Bartholomay, A. (2015). 2014-2015 STEM pathways evaluation. Saint Paul, MN: 
Wilder Research. 


Hakuta, K., Santos, M., & Fang, Z. (2013). Challenges and Opportunities for Language Learning in the Context of the CCSS 
and the NGSS. Journal of Adolescent & Adult Literacy, 56(6), 451-454. 


Harlen, W. (Ed.) (2015). Working with big ideas in science education. Trieste, Italy: Science Education Programme of IAP. 


Klieger, A., & Yakobovitch, A. (2011). Perception of Science Standards’ Effectiveness and Their Implementations by Science 
Teachers. Journal of Science Education and Technology, 20(3), 286-299. 


Michaels, S., & O'Connor, C. (2012). Talk science primer. Cambridge, MA: TERC. 


Michaels, S., Shouse, A.W., & Schweingruber, H.A. (2008). Ready, Set, Science! Putting Research to Work in K-8 Science 
Classrooms. Board on Science Education, Center for Education, Division of Behavioral and Social Sciences and Education. 
Washington, DC: The National Academies Press. 


No. of 
Respondents 
Who Shared It 
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11. | Moulding, B., Bybee, R., & Paulson, N. (2015). A vision and plan for science teaching and learning. Salt Lake City, UT: 
Essential Teaching & Learning PD, LLC. 


Nadelson, L., Seifert, A., & Hendricks, J.K. (2015). Are we preparing the next generation? K-12 teacher knowledge and 
engagement in teaching core STEM practices. ASEE Annual Conference and Exposition, Conference Proceedings, 122nd 
ASEE Annual Conference and Exposition: Making Value for Society. 


National Academies of Sciences, Engineering, and Medicine. (2015). Science Teachers’ Learning: Enhancing Opportunities, 
Creating Supportive Contexts. Committee on Strengthening Science Education through a Teacher Learning Continuum. 
Board on Science Education and Teacher Advisory Council, Division of Behavioral and Social Science and Education. 
Washington, DC: The National Academies Press. 


National Research Council. (2007). Taking Science to School: Learning and Teaching Science in Grades K-8. Committee on 
Science Learning, Kindergarten Through Eighth Grade. Richard A. Duschl, Heidi A. Schweingruber, and Andrew W. Shouse, 
Editors. Board on Science Education, Center for Education. Division of Behavioral and Social Sciences and Education. 
Washington, DC: The National Academies Press. 


National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core 
ideas. Committee on a Conceptual Framework for New K-12 Science Education Standards. Board on Science Education, 
Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. 


National Research Council. (2014). Developing Assessments for the Next Generation Science Standards. Committee on 
Developing Assessments of Science Proficiency in K-12. Board on Testing and Assessment and Board on Science Education, 
James W. Pellegrino, Mark R. Wilson, Judith A. Koenig, and Alexandra S. Beatty, Editors. Division of Behavioral and Social 
Sciences and Education. Washington, DC: The National Academies Press. 


National Research Council. (2015). Guide to Implementing the Next Generation Science Standards. Committee on Guidance 
on Implementing the Next Generation Science Standards. Board on Science Education, Division of Behavioral and Social 
Sciences and Education, Washington, DC: The National Academies Press. 


Next Generation Science Standards. Retrieved from: https://www.achieve.org/next-generation-science-standards 


Osborne, J. & Patterson, A. (2011). Scientific Argument and Explanation: A necessary Distinction? Science Education, 95, 
627-638. 


Pearson, D., Moje, E., & Greenleaf, C. (2010). Literacy and Science: Each in the Service of the Other. Science, 328(5977), 
459-463. 
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21. | Penuel, W.R. & Shepard, L.A. (2016). Assessment and teaching. In D.H. Gitomer & C.A. Bell (Eds.), Handbook of research on 
teaching (pp. 787-850). Washington, DC: AERA. 


22. | Plutzer, E., McCaffrey, M., Hannah, A., Rosenau, J., Berbeco, M., & Reid, A. (2016). Climate confusion among U.S. teachers. 
Science, 351(6274), 664-665. 


Corwin/Sage. 


24. | Project 2061. Retrieved from https://www.aaas.org/program/project2061 


Reiser, B. (2013). What professional development strategies are needed for successful implementation of the Next 


Popham, J. (2016). The ABCs of Educational Testing. Demystifying the Tools that Shape Our Schools. Thousand Oaks, CA: 


Generation Science Standards? Paper presented at the Research Symposium on Science Assessment, Educational Testing 
Service, Washington, DC. 


26. | Reiser, B. (2014). Designing coherent storylines aligned with NGSS for the K-12 classrooms. Presented at the NSELA 
conference. 


Shepard, L., Penuel, W., & Davidson, K. (2017). Design principles for new systems of assessment. Phi Delta Kappa 
International, 98(6), 47-52. 


Sibenaller, J. (n.d.). Chemical reactions: Investigating Exothermic and Endothermic Reactions. Retrieved from 
https://serc.carleton.edu/20203 (Carleton College, MN) 


Research in Science Teaching, 38(9), 1015-1028. 


31. | Wertheim, J., Osborne, J., Quinn, H., Pecheone, R., Schultz, S., Holthuis, N., & Martin, P. (2016). An analysis of existing 
science assessments and the implications for developing assessment tasks for the NGSS. Stanford, CA: SCALE. 


Tate, W. (2001). Science Education as a Civil Right: Urban Schools and Opportunity-to-Learn Considerations. Journal of 
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